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Status of This Memo 


This memo provides information for the Internet community. It does 
not specify an Internet standard of any kind. Distribution of this 
memo is unlimited. 


Abstract 


The Internet Small Computer System Interface (iSCSI) is a SCSI 
transport protocol that maps the SCSI family of application protocols 
onto TCP/IP. Datamover Architecture for iSCSI (DA) defines an 
abstract model in which the movement of data between iSCSI end nodes 
is logically separated from the rest of the iSCSI protocol in order 
to allow iSCSI to adapt to innovations available in new IP 
transports. While DA defines the architectural functions required of 
the class of Datamover protocols, it does not define any specific 
Datamover protocols. Each such Datamover protocol, defined in a 
Separate document, provides a reliable transport for all iSCSI PDUs, 
but actually moves the data required for certain iSCSI PDUs without 
involving the remote iSCSI layer itself. This document begins with 
an introduction of a few new abstractions, defines a layered 
architecture for iSCSI and Datamover protocols, and then models the 
interactions within an iSCSI end node between the iSCSI layer and the 
Datamover layer that happen in order to transparently perform remote 
data movement within an IP fabric. It is intended that this 
definition will help map iSCSI to generic Remote Direct Memory Access 
(RDMA)-capable IP fabrics in the future comprising TCP, the Stream 
Control Transmission Protocol (SCTP), and possibly other underlying 
network transport layers, such as InfiniBand. 
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Motivation 
Intent 


There are relatively new standard protocols that enable Remote Direct 
Memory Access (RDMA) and Remote Direct Data Placement (RDDP) 
technologies to work over IP fabrics. The principal value 
proposition of these technologies is that they enable one end node to 
place data in the final intended buffer on the remote end node, thus 
eliminating the need for a receive path data copy that moves the data 
to its final location. The data copy avoidance in turn eliminates 
unnecessary memory bandwidth consumption, substantially decreases the 
reassembly buffer size requirements, and preserves CPU cycles that 
would otherwise be spent in copying. 


The iSCSI specification [RFC3720] defines a very detailed data 
transfer model that employs SCSI Data-In PDUs, SCSI Data-Out PDUSs, 
and R2T PDUs, in addition to the SCSI Command and SCSI Response PDUs 
that respectively create and conclude the task context for the data 
transfer. In the traditional iSCSI model, the iSCSI protocol layer 
plays the central role in pacing the data transfer and carrying out 
the ensuing data transfer itself. An alternative architecture would 
be for iSCSI to delegate a large part of this data transfer role to a 
Separate protocol layer exclusively designed to move data, which in 
turn is possibly aided by a data movement and placement technology 
such as RDMA. 


If iSCSI were operating in such RDMA environments, iSCSI would be 
shielded from the low-level data transfer mechanics but would only be 
privy to the conclusion of the requested data transfer. Thus, there 
would be an effective "off-loading" of the work that an iSCSI 
protocol layer is expected to perform, compared to today's iSCSI end 
nodes. For such RDMA environments, it is highly desirable that there 
be a standard architecture to separate the data movement part of the 
iSCSI protocol definition from the rest of the iSCSI functionality. 
This architecture precisely defines what a Datamover layer is and 
also describes the model of interactions between the iSCSI layer and 
the Datamover layer (Section 6). In order to satisfy this need, this 
document presents a Datamover Architecture for iSCSI (DA) and 
summarizes a reasonable model for interactions between the iSCSI 
layer and the Datamover layer for each of the iSCSI PDUs that are 
defined in [RFC3720]. Note that while DA is motivated by the advent 
of RDMA over TCP/IP technology, the architecture is not dependent on 
RDMA in its design. DA is intended to be a generic architectural 
framework for allowing different types of Datamovers based on 
different types of RDMA and transport protocols. Adoption of this 
model will help iSCSI proliferate into more environments. 
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1.2. Interpretation of Requirements 


This document introduces certain architectural abstractions and 
builds an abstract functional interface model between iSCSI and 
Datamover protocol layers based on those abstractions. This 
architectural style is motivated by the following desires: 


a) Provide guidance to Datamover protocol designers with respect 
to the functional boundary between iSCSI and the Datamover 
protocols. This guidance is critical since a significant part 
of the [RFC3720] protocol definition is left unchanged by DA 
architecture and the iSCSI notions from [RFC3720] (e.g., tasks, 
ITTs) are leveraged by the Datamover protocol. 


b) Aid existing iSCSI implementations to rapidly adapt to DA 
architecture, largely by leveraging the architectural 
abstractions into implementation constructs -- e.g., functions, 
APIs, modules. 


However, note that DA architecture does not intend to impose any 
implementation specifics per se. When a DA architectural concept 
(e.g., Operational Primitive) is described as mandatory ("MUST") or 
recommended ("SHOULD") of a layer (iSCSI or Datamover) in this 
document, the intent is that an implementation respectively MUST or 
SHOULD produce the same protocol action as what the model describes. 
Specifically, no implementation compliance in terms of names, modules 
or API arguments etc. is implied by this Architecture by such use of 
[RFC2119] terms, only a functional compliance is sought. 


2. Definitions and Acronyms 
2.1. Definitions 


I/O Buffer - A buffer that is used in a SCSI Read or Write operation 
So that SCSI data may be sent from or received by the buffer. 


Datamover protocol - A Datamover protocol is a data transfer wire 
protocol for iSCSI that meets the requirements stated in Section 
6. 


Datamover layer - A Datamover layer is a protocol layer within an end 
node that implements the Datamover protocol. 


Datamover-assisted - An iSCSI connection is said to be "Datamover- 


assisted" when a Datamover layer is enabled for moving control and 
data information on that iSCSI connection. 
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The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 
document are to be interpreted as described in [RFC2119]. 


2.2. Acronyms 


Acronym Definition 

DA Datamover Architecture for iSCSI 
DDP Direct Data Placement Protocol 

DI Datamover Interface 

IANA Internet Assigned Numbers Authority 
IETF Internet Engineering Task Force 
I/O Input - Output 

IP Internet Protocol 

TSESI Internet SCSI 

iSER iSCSI Extensions for RDMA 

ITT Initiator Task Tag 

LO Leading Only 

MPA Marker PDU Aligned Framing for TCP 
PDU Protocol Data Unit 

RDDP Remote Direct Data Placement 

RDMA Remote Direct Memory Access 

R2T Ready To Transfer 

R2TSN Ready To Transfer Sequence Number 
RDMA Remote Direct Memory Access 

RDMAP Remote Direct Memory Access Protocol 
RFC Request For Comments 
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SAM SCSI Architecture Model 

SCSI Small Computer Systems Interface 

SN Sequence Number 

SNACK Selective Negative Acknowledgment - also 


Sequence Number Acknowledgement for Data 
TEP. Transmission Control Protocol 
TTT Target Transfer Tag 
3. Architectural Layering of iSCSI and Datamover Layers 


Figure 1 illustrates an example of the architectural layering of 
iSCSI and Datamover layers, in conjunction with a TCP/IP 
implementation of RDMAP/DDP ([DDP]) layers in an iSCSI end node. 

Note that RDMAP/DDP/MPA and TCP protocol layers are shown here only 
as an example, and in reality, DA is completely oblivious to protocol 
layers below the Datamover layer. The RDMAP/DDP/MPA protocol stack 
provides a generic transport service with direct data placement. 
There is no need to tailor the implementation of this protocol stack 
to the specific ULP to benefit from these services. 
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å sro 25 + SCSI application ër * 
SCSI Layer protocols SCSI Layer 
$ E + $2 + 
v v 
=== === 33252 + iSCSI protocol PES + 
iSCSI Layer | (excluding data | iSCSI Layer 
hearse Sapa eS ES movement) SS RRA PS + 
BS, Sashes CESA DL (Datamover Interfacg)---. “+= ==5, 15282 
v v 
eege EE + a Datamover RT Saas + 
Datamover Layer | protocol | Datamover Layer| 
$2 ETE + $2 EEE EN ERE + 
+------- Ho + Ho +—========== + 
| v | | v | 
| +--------------- + | | +----------------- + | 
RDMAP /DDP /MPA RDMAP /DDP /MPA RDMAP /DDP /MPA 
Layers protocols Layers 
| +--------------- + | | +----------------- + 
| | network | | | network | 
| | transport | | | transport | 
v v 
$ + $ A E ATE N + 
|| TCP Layer || TCP protocol | | TCP Layer | 
| +--------------- + | | +---------------- + | 
Ho +---------- + Ho +----------- + 
AX A sr + 
Figure 1. Datamover Architecture Diagram, 
with the RDMAP Example 
The scope of this document is limited to: 
1. Defining the notion of a Datamover layer and a Datamover 
protocol (Section 6). 
2. Defining the functionality distribution between the iSCSI layer 


and the Datamover layer, along with the communication model 
between the two (Operational Primitives). 


Chadalapaka, et al. Informational [Page 8] 


RFC 5047 DA October 2007 


3. Modeling the interactions between the blocks labeled as "iSCSI 
Layer" and "Datamover Layer" in Figure 1 -- i.e., defining the 
interface labeled "DI" in the figure -- for each defined iSCSI 
PDU, based on the Operational Primitives. 


4. Design Overview 


This document discusses and defines a model for interactions between 
the iSCSI layer and a "Datamover layer" (see Section 6) operating 
within an iSCSI end node, presumably communicating with one or more 
iSCSI end nodes with similar layering. The model for interactions 
for handling different iSCSI operations is called the "Datamover 
Interface" (DI, Section 10), while the architecture itself is called 
the "Datamover Architecture for iSCSI" (DA). It is likely that the 
architecture will have implications on the Datamover wire protocols 
as DA places certain requirements and functionality expectations on 
the Datamover layer. However, this document itself neither defines 
any new wire protocol for the Datamover layer, nor any potential 
modifications to the iSCSI wire protocol to employ the Datamover 
layer. The scope of this document is strictly limited to specifying 
the architectural framework and the minimally required interactions 
that happen within an iSCSI end node to leverage the Datamover layer. 


The design ideas behind DA can be summarized as follows: 


1) DA defines an abstract functional interface model of the iSCSI 
layer's interactions with a Datamover layer below -- i.e., DA 
models the interactions between the logical "bottom" interface 
of iSCSI and the logical "top" interface of a Datamover. 


2) DA guides the wire protocol for a Datamover layer by defining 
the iSCSI knowledge that the Datamover layer may utilize in its 
protocol definition (as an example, this document completely 
limits the notion of "iSCSI session" to the iSCSI layer). 


3) DA is designed to allow implementation of the Datamover layer 
either in hardware or in software. 


4) DA is not a wire protocol spec, but an architecture that also 
models the interactions between iSCSI and Datamover layers 
operating within an iSCSI end node. 


5) DA by design seeks to model the iSCSI-Datamover interactions in 
a way that the modeling is independent of the specifics of 
either a particular iSCSI revision or an instantiation of a 
Datamover layer. 
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6) DA introduces and relies on the notion of a defined set of 
Operational Primitives (could be seen as entry point 
definitions in implementation terms) provided by each layer to 
the other to carry out the request-response interactions. 


7) DA is intended to allow Datamover protocol definitions with 
minimal changes to existing iSCSI implementations. 


8) DA is designed to allow the iSCSI layer to completely rely on 
the Datamover layer for all data transport needs. 


9) DA models the architecturally required minimal interactions 
between an operational iSCSI layer and a Datamover layer to 
realize the iSCSI-transparent data movement. There may be 
several other interactions in a typical implementation in order 
to bootstrap a Datamover layer (or an iSCSI layer) into 
operation, but they are outside the scope of this document. 


Note that in summary, DA is architected to support many different 
Datamover protocols operating under the iSCSI layer. One such 
example of a Datamover protocol is iSER [iSER]. 


5. Architectural Concepts 
5.1. iSCSI PDU Types 


This section defines the iSCSI PDU classification terminology, as 
defined and used in this document. Out of the set of legal iSCSI 
PDUs defined in [RFC3720], as we will see in Section 5.1.1, the iSCSI 
layer does not request a SCSI Data-Out PDU carrying solicited data 
for transmission across the Datamover Interface per this 
architecture. For this reason, the SCSI Data-Out PDU carrying 
solicited data is excluded in the iSCSI PDU classification we 
introduce in this section (for SCSI Data-Out PDUs for unsolicited 
Data, see Section 5.1.2). The rest of the legal iSCSI PDUs that may 
be exchanged across the Datamover Interface are defined to consist of 
two classes: 


1) iSCSI data-type PDUs 
2) iSCSI control-type PDUs 
5.1.1.  iSCSI Data-Type PDUs 
An iSCSI data-type PDU is defined as an iSCSI PDU that causes data 
transfer, transparent to the remote iSCSI layer, to take place 


between the peer iSCSI nodes on a Full Feature Phase iSCSI 
connection. A data-type PDU, when requested for transmission by the 
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sender iSCSI layer, results in the associated data transfer without 

the participation of the remote iSCSI layer, i.e., the PDU itself is 
not delivered as-is to the remote iSCSI layer. The following iSCSI 

PDUs constitute the set of iSCSI data-type PDUs: 


1) SCSI Data-In PDU 
2) R2T PDU 


In an iSCSI end node structured as an iSCSI layer and a Datamover 
layer as defined in this document, the solicitation for Data-Out 
(i.e., R2T PDU) is not delivered to the initiator iSCSI layer, per 
the definition of an iSCSI data-type PDU. The data transfer is 
instead performed via the mechanisms known to the Datamover layer 
(e.g., RDMA Read). This in turn implies that a SCSI Data-Out PDU for 
Solicited data is never requested for transmission across the 
Datamover Interface at the initiator. 


5.1.2. iSCSI Control-Type PDUs 


Any iSCSI PDU that is not an iSCSI data-type PDU and also not a 
solicited SCSI Data-Out PDU is defined as an iSCSI control-type PDU. 
Specifically, note that SCSI Data-Out PDUs for unsolicited Data are 
defined as iSCSI control-type PDUs. 


5.2. Data Descriptor 


A Data Descriptor is an information element that describes an 
iSCSI/SCSI data buffer, provided by the iSCSI layer to its local 
Datamover layer or provided by the Datamover layer to its local iSCSI 
layer for identifying the data associated respectively with the 
requested or completed operation. 


In implementation terms, a Data Descriptor may be a scatter-gather 
list describing a local buffer, the exact structure of which is 
subject to the constraints imposed by the operating environment on 
the local iSCSI node. 


5.3. Connection Handle 


A Connection Handle is an information element that identifies the 
particular iSCSI connection for which an inbound or outbound iSCSI 
PDU is intended. A connection handle is unique for a given pair of 
an iSCSI layer instance and a Datamover layer instance. The 
Connection Handle qualifier is used in all invocations of any 
Operational Primitive for connection identification. 
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Note that the Connection Handle is conceptually different from the 
Connection Identifier (CID) defined by the iSCSI specification. 

While the CID is a unique identifier of an iSCSI connection within an 
iSCSI session, the uniqueness of the Connection Handle extends to the 
entire iSCSI layer instance coupled with the Datamover layer 
instance, across possibly multiple iSCSI sessions. 


In implementation terms, a Connection Handle could be an opaque 
identifier exchanged between the iSCSI layer and the Datamover layer 
at the connection login time. One may also consider it to be similar 
in scope of uniqueness to a socket identifier. The exact structure 
and modalities of exchange of a Connection Handle between the two 
layers is implementation-specific. 


5.4. Operational Primitive 


An Operational Primitive, in this document, is an abstract functional 
interface procedure that requests another layer perform a specific 
action on the requestor's behalf or notifies the other layer of some 
event. The Datamover Interface between an iSCSI layer instance and a 
Datamover layer instance within an iSCSI end node uses a set of 
Operational Primitives to define the functional interface between the 
two layers. Note that not every invocation of an Operational 
Primitive may elicit a response from the requested layer. This 
document describes the types of Operational Primitives that are 
implicitly required and provided by the iSCSI protocol layer as 
defined in [RFC3720], and the semantics of these Primitives. 


Note that ownership of buffers and data structures is likely to be 
exchanged between the iSCSI layer and its local Datamover layer in 
invoking the Operational Primitives defined in this architecture. 
The buffer management details, including how buffers are allocated 
and released, are implementation-specific and thus are outside the 
Scope of this document. 


Each Operational Primitive invocation needs a certain "information 
context" (e.g., Connection Handle) for performing the specific action 
being requested. The required information context is described in 
this document by a listing of "qualifiers" on each invocation, in the 
style of function call arguments. There is no specific 
implementation implied in this notation. The "qualifiers" of any 
Operational Primitive invocation specified in this document thus 
represent the mandatory information context that the Operational 
Primitive invocation MUST consider in performing the action. While 
the qualifiers are required, the method of realizing the qualifiers 
(passed synchronously with invocation, or retrieved from task 
context, or retrieved from shared memory etc.) is really up to the 
implementations. 


Chadalapaka, et al. Informational [Page 12] 


RFC 5047 DA October 2007 


When an Operational Primitive implementation is described as 
mandatory ("MUST") or recommended ("SHOULD") of a layer (iSCSI or 
Datamover) in this document, the intent is that an implementation 
respectively MUST or SHOULD produce the same protocol action as what 
the model describes. 


5.5. Transport Connection 


The term "Transport Connection" is used in this document as a generic 
term to represent the end-to-end logical connection as defined by the 
underlying reliable transport protocol. For this document, all 
instances of Transport Connection refer to a TCP connection. 


6. Datamover Layer and Datamover Protocol 


This section introduces the notion of a "Datamover layer" and 
"Datamover protocol" as meant in this document, and defines the 
requirements on a Datamover protocol. 


A Datamover layer is the implementation component that realizes a 
Datamover protocol functionality in an iSCSI-capable end node in 
communicating with other iSCSI end nodes with similar capabilities. 
More specifically, a "Datamover layer" MUST provide the following 
functionality and the "Datamover protocol" MUST consist of the wire 
protocol required to realize the following functionality: 


1) guarantee that all the necessary data transfers take place when 
the local iSCSI layer requests transmitting a command (in order 
to complete a SCSI command, for an initiator), or 
sending/receiving an iSCSI data sequence (in order to complete 
part of a SCSI command for a target). 


2) transport an iSCSI control-type PDU as-is to the peer Datamover 
layer when requested to do so by the local iSCSI layer. 


3) provide notification and delivery to the iSCSI layer upon 
arrival of an iSCSI control-type PDU. 


4) provide an initiator-to-target data acknowledgement of SCSI 
read data back to the target iSCSI layer, when requested. 


5) provide an asynchronous notification upon completion of a 
requested data transfer operation that moved data without 
involving the iSCSI layer. 


6) place the SCSI data into the I/O buffers or pick up the SCSI 


data for transmission out of the data buffers that the iSCSI 
layer had requested to be used for a SCSI I/O. 
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7) provide an error-free (i.e., must have at least the same level 
of assurance of data integrity as the CRC32C iSCSI data 
digest), reliable, in-order delivery transport mechanism over 
IP networks in performing the data transfer, and asynchronously 
notify the iSCSI layer upon iSCSI connection termination. 


Note that this architecture expects that each compliant Datamover 
protocol will define the precise means of satisfying the requirements 
Specified in this section. 


In order to meet the functional requirements listed in this section, 
certain Datamover protocols may require pre-posted buffers from the 
local iSCSI protocol layer via mechanisms outside the scope of this 
document. In some implementations, the absence of such buffers may 
result in a connection failure.  Datamover protocols may also realize 
these functional requirements via methods not explicitly listed in 
this document. 


7. Functional Overview 


This section presents an overview of the functional interactions 
between the iSCSI layer and the Datamover layer as intended by this 
Architecture. 


Tels Startup 


The iSCSI Login Phase on an iSCSI connection occurs as defined in 
[RFC3720]. The Architecture assumes that at the end of the Login 
Phase, both the initiator and target, if they had so decided, 
transition the connection to being Datamover-assisted. The precise 
means of how an iSCSI initiator and an iSCSI target agree on having 
the connection Datamover-assisted is defined by the Datamover 
protocol. The only architectural requirement is that all iSCSI 
interactions in the iSCSI Full Feature Phase MUST be Datamover- 
assisted subject to the prior agreement, meaning that the Datamover 
protocol is in the iSCSI-to-iSCSI communication path below the iSCSI 
layer on either side as shown in Figure 1. DA defines the 

Enable Datamover Operational Primitive (Section 8.6) to bring about 
this transition to a Datamover-assisted connection. 


The Architecture also assumes that the Datamover layer may require a 
certain number of opaque local resources for making a connection 
Datamover-assisted. DA thus defines the 

Allocate Connection Resources Operational Primitive (Section 8.4) to 
model this interaction. This Primitive is intended to be invoked on 
each side once the two sides decide (as previously noted) to have the 
connection be Datamover-assisted. The expected sequence of Primitive 
invocations is depicted in Figures 2 and 3 in Section 13.2. Figures 
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4, 5, and 6 illustrate how the Primitives may be employed to deal 
with various legal login outcomes. 


7.2. Full Feature Phase 


All iSCSI peer communication in the Full Feature Phase happens 
through the Datamover layers if the iSCSI connection is Datamover- 
assisted. The Architecture assumes that a Datamover layer may 
require a certain number of opaque local resources for each new iSCSI 
task. In the normal course of execution, these task-level resources 
in the Datamover layer are assumed to be transparently allocated on 
each task initiation and deallocated on the conclusion of each task 
as appropriate. In exception scenarios however -- scenarios that do 
not yield a SCSI Response for each task such as ABORT TASK operation 
-- the Architecture assumes that the Datamover layer needs to be 
notified of the individual task terminations to aid its task-level 
resource management. DA thus defines the Deallocate Task Resources 
Operational Primitive (Section 8.9) to model this task-resource 
management. In specifying the ITT qualifier for the 
Deallocate Task Resources Primitive, the Architecture further assumes 
that the Datamover layer tracks its opaque task-level local resources 
by the iSCSI ITT. DA also defines Send Control (Section 8.1), 

Put Data (Section 8.2), Get Data (Section 8.3), 

Data Completion Notify (Section 9.3), Data ACK Notify (Section 9.4), 
and Control Notify (Section 9.1) Operational Primitives to model the 
various Full Feature Phase interactions. 


Figures 9, 10, and 11 in Section 13.2 show some Full Feature Phase 
interactions -- SCSI Write task, SCSI Read task, and a SCSI Read Data 
acknowledgement, respectively. Figure 12 in Section 13.2 illustrates 
how an ABORT TASK operation can be modeled leading to deterministic 
resource cleanup on the Datamover layer. 


7.3. Wrap-up 


Once an iSCSI connection becomes Datamover-assisted, the connection 
continues in that state until the end of the Full Feature Phase, 
i.e., the termination of the connection. The Architecture assumes 
that when a connection is normally logged out, the Datamover layer 
needs to be notified so that its connection-level opaque resources 
(see Section 7.1) may be freed up. DA thus defines a 

Connection Terminate Operational Primitive (Section 8.7) to model 
this interaction. The Architecture further assumes that when a 
connection termination happens without iSCSI layer's involvement 
(e.g., TCP RST), the Datamover layer is capable of locally cleaning 
up its task-level and connection-level resources before notifying the 
iSCSI layer of the fact. DA thus defines the 
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Connection Terminate Notify Operational Primitive (Section 9.2) to 
model this interaction. 

Figures 7 and 8 in Section 13.2 illustrate the interactions between 
the iSCSI and Datamover layers in normal and unexpected connection 
termination scenarios. 

8. Operational Primitives Provided by the Datamover Layer 
While the iSCSI specification itself does not have a notion of 
Operational Primitives, any iSCSI layer implementing the iSCSI 
Specification functionally requires the following Operational 
Primitives from its Datamover layer. Thus, any Datamover protocol 
compliant with this architecture MUST implement the Operational 
Primitives described in this section. These Operational Primitives 
are invoked by the iSCSI layer as appropriate. Unless otherwise 
stated, all the following Operational Primitives may be used both on 
the initiator side and the target side. In general programming 
terminology, this set of Operational Primitives may be construed as 
"down calls". 

1) Send Control 

2) Put Data 

3) Get Data 

4) Allocate Connection Resources 
5) Deallocate Connection Resources 
6) Enable Datamover 

7) Connection Terminate 

8) Notice Key Values 

9) Deallocate Task Resources 

8.1. Send Control 
Input qualifiers: Connection Handle, iSCSI PDU-specific qualifiers 
Return Results: Not specified. 

An iSCSI layer requests that its local Datamover layer transmit an 


iSCSI control-type PDU to the peer iSCSI layer operating in the 
remote iSCSI node by this Operational Primitive. The Datamover layer 


Chadalapaka, et al. Informational [Page 16] 


RFC 5047 DA October 2007 


performs the requested operation, and may add its own protocol 
headers in doing so. The iSCSI layer MUST NOT invoke the 

Send Control Operational Primitive on an iSCSI connection that is not 
yet Datamover-assisted. 


An initiator iSCSI layer requesting the transfer of a SCSI Command 
PDU or a target iSCSI layer requesting the transfer of a SCSI 
response PDU are examples of invoking the Send Control Operational 
Primitive. As Section 10.3.1 illustrates later on, the iSCSI PDU- 
specific qualifiers in this example are: BHS and AHS, 
DataDescriptorOut, DataDescriptorIn, ImmediateDataSize, and 
UnsolicitedDataSize. 


8.2. Put Data 


Input qualifiers: Connection Handle, contents of a SCSI Data-In PDU 
header, Data Descriptor, Notify Enable 


Return Results: Not specified. 


An iSCSI layer requests that its local Datamover layer transmit the 
data identified by the Data Descriptor for the SCSI Data-In PDU to 
the peer iSCSI layer on the remote iSCSI node by this Operational 
Primitive. The Datamover layer performs the operation by using its 
own protocol means, completely transparent to the remote iSCSI layer. 
The iSCSI layer MUST NOT invoke the Put Data Operational Primitive on 
an iSCSI connection that is not yet Datamover-assisted. 


The Notify Enable qualifier is used to request the local Datamover 
layer to generate or not generate the eventual local completion 
notification to the iSCSI layer for this Put Data invocation. For 
detailed semantics of this qualifier, see Section 9.3. 


A Put Data Primitive may only be invoked by an iSCSI layer on the 
target to its local Datamover layer. 


A target iSCSI layer requesting the transfer of an iSCSI read data 
sequence (also known as a read burst) is an example of invoking the 
Put Data Operational Primitive. 


8.3. Get Data 


Input qualifiers: Connection Handle, contents of an R2T PDU, 
Data Descriptor, Notify Enable 


Return Results: Not specified. 
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An iSCSI layer requests that its local Datamover layer retrieve 
certain data identified by the R2T PDU from the peer iSCSI layer on 
the remote iSCSI node and place it into the buffer identified by the 
Data Descriptor by invoking this Operational Primitive. The 
Datamover layer performs the operation by using its own protocol 
means, completely transparent to the remote iSCSI layer. The iSCSI 
layer MUST NOT invoke the Get Data Operational Primitive on an iSCSI 
connection that is not yet Datamover-assisted. 


The Notify Enable qualifier is used to request that the local 
Datamover layer generate or not generate the eventual local 
completion notification to the iSCSI layer for this Get Data 
invocation. For detailed semantics of this qualifier, see Section 
9.3. 


A Get Data Primitive may only be invoked by an iSCSI layer on the 
target to its local Datamover layer. 


A target iSCSI layer requesting the transfer of an iSCSI write data 
sequence (also known as a write burst) is an example of invoking the 
Get Data Operational Primitive. 


8.4.  Allocate Connection Resources 
Input qualifiers: Connection Handle[, Resource Descriptor ] 
Return Results: Status. 


By invoking this Operational Primitive, an iSCSI layer requests that 
its local Datamover layer perform all the Datamover-specific resource 
allocations required for the Full Feature Phase of an iSCSI 
connection. The Connection Handle identifies the connection for 
which the iSCSI layer is requesting resources to be allocated. 
Allocation of these resources is a step towards eventually 
transitioning the connection to become a Datamover-assisted iSCSI 
connection. Note that the Datamover layer however does not allocate 
any Datamover-specific task-level resources upon invocation of this 
Primitive. 


An iSCSI layer, in addition, optionally specifies the 
implementation-specific resource requirements for the iSCSI 
connection to the Datamover layer by passing an input qualifier 
called Resource Descriptor. The exact structure of a 

Resource Descriptor is implementation-dependent, and hence 
structurally opaque to DA. 


A return result of Status-success means that the 
Allocate Connection Resources invocation corresponding to that 
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Connection Handle succeeded. If an Allocate Connection Resources 
invocation is made for a Connection Handle for which an earlier 
invocation succeeded, the return Status must be success and the 
request will be ignored by the Datamover layer. A return result of 
Status-failure means that the Allocate Connection Resources 
invocation corresponding to that Connection Handle failed. There 
MUST NOT be more than one Allocate Connection Resources Primitive 
invocation outstanding for a given Connection Handle at any time. 


The iSCSI layer must invoke the Allocate Connection Resources 
Primitive before the invocation of the Enable Datamover Primitive. 


8.5. Deallocate Connection Resources 
Input qualifiers: Connection Handle 
Return Results: Not specified. 


By invoking this Operational Primitive, an iSCSI layer requests that 
its local Datamover layer deallocate all the Datamover-specific 
resources that may have been allocated earlier for the Transport 
Connection identified by the Connection Handle. The iSCSI layer may 
invoke this Operational Primitive when the Datamover-specific 
resources associated with the Connection Handle are no longer 
necessary (such as the Login failure of the corresponding iSCSI 
connection). 


8.6. Enable Datamover 


Input qualifiers: Connection Handle, Transport Connection Descriptor 
[, Final Login Response PDU] 


Return Results: Not specified. 


By invoking this Operational Primitive, an iSCSI layer requests that 
its local Datamover layer assist all further iSCSI exchanges on the 
iSCSI connection (i.e., to make the connection Datamover-assisted) 
identified by the Connection Handle, for which the Datamover-specific 
resource allocation was earlier made. The iSCSI layer MUST NOT 
invoke the Enable Datamover Operational Primitive for an iSCSI 
connection unless there is a corresponding prior resource allocation. 


The Final Login Response PDU input qualifier is applicable only for a 
target, and contains the final Login Response that concludes the 
iSCSI Login Phase and which must be sent as a byte stream as expected 
by the initiator iSCSI layer. When this qualifier is used, the 
target-Datamover layer MUST transmit this final Login Response before 
Datamover assistance is enabled for the Transport Connection. 
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The iSCSI layer identifies the specific Transport Connection 
associated with the Connection Handle to the Datamover layer by 
Specifying the Transport Connection Descriptor. The exact structure 
of this Descriptor is implementation-dependent. 


8.7. Connection Terminate 
Input qualifiers: Connection Handle 
Return Results: Not specified. 


By invoking this Operational Primitive, an iSCSI layer requests that 
its local Datamover layer terminate the Transport Connection and 
deallocate all the connection and task resources associated with the 
Connection Handle. When this Operational Primitive invocation 
returns to the iSCSI layer, the iSCSI layer may assume the full 
ownership of all the iSCSI-level resources, e.g., I/O Buffers, 
associated with the connection. This Operational Primitive may be 
invoked only with a valid Connection Handle, and the Transport 
Connection associated with the Connection Handle must already be 
Datamover-assisted. 


8.8. Notice Key Values 


Input qualifiers: Connection Handle, Number of keys, a list of Key- 
Value pairs. 


Return Results: Not specified. 


By invoking this Operational Primitive, an iSCSI layer requests that 
its local Datamover layer take note of the negotiated values of the 
listed keys for the Transport Connection. This Operational Primitive 
may be invoked only with a valid Connection Handle, and the Key-Value 
pairs MUST be the current values that were successfully agreed upon 
by the iSCSI peers for the connection. The Datamover layer may use 
the values of the keys to aid the Datamover operation as it deems 
appropriate. The specific keys to be passed as input qualifiers and 
the point(s) in time this Operational Primitive is invoked are 
implementation-dependent. 


8.9. Deallocate Task Resources 
Input qualifiers: Connection Handle, ITT 
Return Results: Not specified. 


By invoking this Operational Primitive, an iSCSI layer requests that 
its local Datamover layer deallocate all Datamover-specific resources 
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that earlier may have been allocated for the task identified by the 
ITT qualifier. The iSCSI layer uses this Operational Primitive 
during exception processing when one or more active tasks are to be 
terminated without corresponding SCSI Response PDUs. This Primitive 
MUST be invoked for each active task terminated without a SCSI 
Response PDU. This Primitive MUST NOT be invoked by the iSCSI layer 
when a SCSI Response PDU normally concludes a task. When a SCSI 
Response PDU normally concludes a task (even if the SCSI Status was 
not a success), the Datamover layer is assumed to have automatically 
deallocated all Datamover-specific task resources for that task. 
Refer to Section 7.2 for a related discussion on the Architectural 
assumptions on the task-level Datamover resource management, 
especially with respect to when the resources are assumed to be 
allocated. 


9. Operational Primitives Provided by the iSCSI Layer 


While the iSCSI specification itself does not have a notion of 
Operational Primitives, any iSCSI layer implementing the iSCSI 
specification would have to provide the following Operational 
Primitives to its local Datamover layer. Thus, any iSCSI protocol 
implementation compliant with this architecture MUST implement the 
Operational Primitives described in this section. These Operational 
Primitives are invoked by the Datamover layer as appropriate and when 
the iSCSI connection is Datamover-assisted. Unless otherwise stated, 
all the following Operational Primitives may be used both on the 


initiator side and the target side. In general programming 
terminology, this set of Operational Primitives may be construed as 
"up calls". 


1) Control Notify 
2) Connection Terminate Notify 
3) Data Completion Notify 
4) Data ACK Notify 
9.1. Control Notify 
Input qualifiers: Connection Handle, an iSCSI control-type PDU. 
Return Results: Not specified. 
A Datamover layer notifies its local iSCSI layer, via this 
Operational Primitive, of the arrival of an iSCSI control-type PDU 


from the peer Datamover layer on the remote iSCSI node. The iSCSI 
layer processes the control-type PDU as defined in [RFC3720]. 
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A target iSCSI layer being notified of the arrival of a SCSI command 
is an example of invoking the Control Notify Operational Primitive. 


Note that implementations may choose to describe the "iSCSI control- 
type PDU" qualifier in this notification using a Data Descriptor 
(Section 5.2) and not necessarily one contiguous buffer. 


9.2. Connection Terminate Notify 
Input qualifiers: Connection Handle 
Return Results: Not specified. 


A Datamover layer notifies its local iSCSI layer on an unsolicited 
termination or failure of an iSCSI connection providing the 
Connection Handle associated with the iSCSI Connection. The iSCSI 
layer MUST consider the Connection Handle to be invalid upon being so 
notified. The iSCSI layer processes the connection termination as 
defined in [RFC3720]. The Datamover layer MUST deallocate the 
connection and task resources associated with the terminated 
connection before notifying the iSCSI layer of the termination via 
this Operational Primitive. 


A target iSCSI layer is notified of an ungraceful connection 
termination by the Datamover layer when the underlying Transport 
Connection is torn down. Such a Connection Terminate Notify 
Operational Primitive may be triggered, for example, by a TCP RESET 
in cases where the underlying Transport Connection uses TCP. 


9.3. Data Completion Notify 
Input qualifiers: Connection Handle, ITT, SN 
Return Results: Not specified. 


A Datamover layer notifies its local iSCSI layer on completing the 
retrieval of the data or upon sending the data, as requested in a 
prior iSCSI data-type PDU, from/to the peer Datamover layer on the 
remote iSCSI node via this Operational Primitive. The iSCSI layer 
processes the operation as defined in [RFC3720]. 


SN may be either the DataSN associated with the SCSI Data-In PDU or 
R2TSN associated with the R2T PDU depending on the SCSI operation. 
Note that, for targets, a TTT (see [RFC3720]) could have been 
Specified instead of an SN. However, the considered choice was to 
leave the SN to be the qualifier for two reasons -- a) it is generic 
and applicable to initiators and targets as well as Data-In and 
Data-Out, and b) having both SN and TTT qualifiers for the 
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10. 


10. 


notification is considered onerous on the Datamover layer, in terms 
of state maintenance for each completion notification. The 
implication of this choice is that iSCSI target implementations will 
have to adapt to using the ITT-SN tuple in associating the solicited 
data to the appropriate task, rather than the ITT-TTT tuple for doing 
the same. 


If Notify Enable is set in either a Put Data or a Get Data 
invocation, the Datamover layer MUST invoke the 

Data Completion Notify Operational Primitive upon completing that 
requested data transfer. If the Notify Enable was cleared in either 
a Put Data or a Get Data invocation, the Datamover layer MUST NOT 
invoke the Data Completion Notify Operational Primitive upon 
completing that requested data transfer. 


A Data Completion Notify invocation serves to notify the iSCSI layer 
of the Put Data or Get Data completion, respectively. As earlier 
noted in Sections 8.2 and 8.3, specific Datamover protocol 
definitions may restrict the usage scope of Put Data and Get Data, 
and thus implicitly the usage scope of Data Completion Notify. 


A target iSCSI layer being notified of the retrieval of a write data 
Sequence is an example of invoking the Data Completion Notify 
Operational Primitive. 


.4. Data ACK Notify 


Input qualifiers: Connection Handle, ITT, DataSN 
Return Results: Not specified. 


A target Datamover layer notifies its local iSCSI layer of the 
arrival of a previously requested data acknowledgement from the peer 
Datamover layer on the remote (initiator) iSCSI node via this 
Operational Primitive. The iSCSI layer processes the data 
acknowledgement notification as defined in [RFC3720]. 


A target iSCSI layer being notified of the arrival of a data 
acknowledgement for a certain SCSI Read data PDU is the only example 
of invoking the Data ACK Notify Operational Primitive. 

Datamover Interface (DI) 
1. Overview 
This section describes the model of interactions between iSCSI and 


Datamover layers when the iSCSI connection is Datamover-assisted so 
the iSCSI layer may carry out the following: 
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- send iSCSI data-type PDUs and exchange iSCSI control-type PDUs, 
and 


- handle asynchronous notifications such as completion of data 
sequence transfer and connection failure. 


This chapter relies on the notion of Operational Primitives (Section 
5.4) to define DI. 


-2. Interactions for Handling Asynchronous Notifications 


.2.1. Connection Termination 


As stated in Section 9.2, the Datamover layer notifies the iSCSI 
layer of a failed or terminated connection via the 

Connection Terminate Notify Operational Primitive. The iSCSI layer 
MUST consider the connection unusable upon the invocation of this 
Primitive and handle the connection termination as specified in 
[RFC3720]. 


.2.2. Data Transfer Completion 


As stated in Section 9.3, the Datamover layer notifies the iSCSI 
layer of a completed data transfer operation via the 

Data Completion Notify Operational Primitive. The iSCSI layer 
processes the transfer completion as specified in [RFC3720]. 


.2.2.1. Completion of a Requested SCSI Data Transfer 


To notify the iSCSI layer of the completion of a requested iSCSI 
data-type PDU transfer, the Datamover layer uses the 

Data Completion Notify Operational Primitive with the following input 
qualifiers. 


a) Connection Handle. 

b) ITT: Initiator Task Tag semantics as defined in [RFC3720]. 

c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN for an 
iSCSI R2T PDU. The semantics for both types of sequence 
numbers are as defined in [RFC3720]. 

The rationale for choosing SN is explained in Section 9.3. 
Every invocation of the Data Completion Notify Operational Primitive 
MUST be preceded by an invocation of the Put Data or Get Data 


Operational Primitive with the Notify Enable qualifier set by the 
iSCSI layer at an earlier point in time. 
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10.2.3. Data Acknowledgement 


[RFC3720] allows the iSCSI targets to optionally solicit data 
acknowledgement from the initiator for one or more Data-In PDUs, via 
setting of the A-bit on a Data-In PDU. The Data ACK Notify 
Operational Primitive with the following input qualifiers is used by 
the target Datamover layer to notify the local iSCSI layer of the 
arrival of data acknowledgement of a previously solicited iSCSI read 
data acknowledgement. This Operational Primitive thus is applicable 
only to iSCSI targets. 


a) Connection Handle. 
b) ITT: Initiator Task Tag semantics as defined in [RFC3720]. 


C) DataSN: of the next SCSI Data-In PDU, which immediately follows 
the SCSI Data-In PDU with the A-bit set to which this 
notification corresponds, with semantics as defined in 
[RFC3720]. 


Every invocation of the Data ACK Notify Operational Primitive MUST be 
preceded by an invocation of the Put Data Operational Primitive by 
the iSCSI target layer with the A-bit set to 1 at an earlier point in 
time. 


10.3. Interactions for Sending an iSCSI PDU 


This section discusses the model of interactions for sending each of 
the iSCSI PDUs defined in [RFC3720]. A Connection Handle (see 
Section 5.3) is assumed to qualify each of these interactions so that 
the Datamover layer can route it to the appropriate Transport 
Connection. The qualifying Connection Handle is not explicitly 
listed in the subsequent sections. 


Note that the defined list of input qualifiers represents the 
semantically required set for the Datamover layer to consider in 
implementing the Primitive in each interaction described in this 
section (see Section 5.4 for an elaboration).  Implementations may 
choose to deduce the qualifiers in ways that are optimized for the 
implementation specifics. Two examples of this are: 


1. For SCSI command (Section 10.3.1), deducing the 
ImmediateDataSize input qualifier from the DataSegmentLength 
field of the SCSI Command PDU. 


2. For SCSI Data-Out (Section 10.3.5.1), deducing the 


DataDescriptorOut input qualifier from the associated SCSI 
command invocation qualifiers (assuming such state is 
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maintained) in conjunction with BHS fields of the SCSI Data-Out 
PDU. 


10.3.1. SCSI Command 


The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a SCSI Command 
PDU. 


a) BHS and AHS, if any, of the SCSI Command PDU as defined in 
[RFC3720]. 


b) DataDescriptorOut: that defines the I/O Buffer meant for Data- 
Out for the entire command, in the case of a write or 
bidirectional command. 


c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In 
for the entire command, in the case of a read or bidirectional 


command. 


d) ImmediateDataSize: that defines the number of octets of 
immediate unsolicited data for a write/bidirectional command. 


e) UnsolicitedDataSize: that defines the number of octets of 
immediate and non-immediate unsolicited data for a 
write/bidirectional command. 

10.3.2. SCSI Response 
The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a SCSI Response 
PDU. 
a) BHS of the SCSI Response PDU as defined in [RFC3720]. 


b) DataDescriptorStatus: that defines the iSCSI buffer that 
contains the sense and response information for the command. 


10.3.3. Task Management Function Request 
The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a Task 


Management Function Request PDU. 


a) BHS of the Task Management Function Request PDU as defined in 
[RFC3720]. 
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b) DataDescriptorOut: that defines the I/O Buffer meant for Data- 
Out for the entire command, in the case of a write or 
bidirectional command. (Only valid if Function-"TASK REASSIGN" 
- [RFC3720].) 


c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In 
for the entire command, in the case of a read or bidirectional 
command. (Only valid if Function-"TASK REASSIGN" - [RFC3720].) 

10.3.4. Task Management Function Response 
The Send Control Operational Primitive with the following input 
qualifier is used for requesting the transmission of a Task 


Management Function Response PDU. 


a) BHS of the Task Management Function Response PDU as defined in 
[RFC3720]. 


10.3.5. SCSI Data-Out and SCSI Data-In 

10.3.5.1. SCSI Data-Out 
The Send Control Operational Primitive with the following input 
qualifiers is used by the initiator iSCSI layer for requesting the 
transmission of a SCSI Data-Out PDU carrying the non-immediate 
unsolicited data. 


a) BHS of the SCSI Data-Out PDU as defined in [RFC3720]. 


b) DataDescriptorOut: that defines the I/O Buffer with the Data- 
Out to be carried in the iSCSI data segment of the PDU. 


10.3.5.2. SCSI Data-In 
The Put Data Operational Primitive with the following input 
qualifiers is used by the target iSCSI layer for requesting the 
transmission of the data carried by a SCSI Data-In PDU. 


a) BHS of the SCSI Data-In PDU as defined in [RFC3720]. 


b) DataDescriptorIn: that defines the I/O Buffer with the Data-In 
being requested for transmission. 
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10.3.6. Ready To Transfer (R2T) 


The Get Data Operational Primitive with the following input 
qualifiers is used by the target iSCSI layer for requesting the 
retrieval of the data as specified by the semantic content of an R2T 
PDU. 


a) BHS of the Ready To Transfer PDU as defined in [RFC3720]. 


b) DataDescriptorOut: that defines the I/O Buffer for the Data-Out 
being requested for retrieval. 


10.3.7. Asynchronous Message 
The Send_Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of an Asynchronous 
Message PDU. 
a) BHS of the Asynchronous Message PDU as defined in [RFC3720]. 


b) DataDescriptorSense: that defines an iSCSI buffer that contains 
the sense and iSCSI Event information. 


10.3.8. Text Request 
The Send_Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a Text Request 
PDU. 
a) BHS of the Text Request PDU as defined in [RFC3720]. 


b) DataDescriptorTextOut: that defines the iSCSI Text Request 
buffer. 


10.3.9. Text Response 
The Send_Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a Text Response 
PDU. 
a) BHS of the Text Response PDU as defined in [RFC3720]. 


b) DataDescriptorTextIn: that defines the iSCSI Text Response 
buffer. 
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10.3.10. Login Request 


The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a Login Request 
PDU. 


a) BHS of the Login Request PDU as defined in [RFC3720]. 


b) DataDescriptorLoginRequest: that defines the iSCSI Login 
Request buffer. 


Note that specific Datamover protocols may choose to disallow the 
standard DA Primitives from being used for the iSCSI Login Phase. 
When used in conjunction with such Datamover protocols, an attempt to 
send a Login Request via the Send Control Operational Primitive 
invocation is clearly an error scenario, as the Login Request PDU is 
being sent while the connection is in the iSCSI Full Feature Phase. 
It is outside the scope of this document to specify the resulting 
implementation behavior in this case -- [RFC3720] already defines the 
error handling for this error scenario. 


10.3.11. Login Response 


The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a Login 
Response PDU. 


a) BHS of the Login Response PDU as defined in [RFC3720]. 


b) DataDescriptorLoginResponse: that defines the iSCSI Login 
Response buffer. 


Note that specific Datamover protocols may choose to disallow the 
standard DA Primitives from being used for the iSCSI Login Phase. 
When used in conjunction with such Datamover protocols, an attempt to 
send a Login Response via the Send Control Operational Primitive 
invocation is clearly an error scenario, as the Login Response PDU is 
being sent while in the iSCSI Full Feature Phase. It is outside the 
Scope of this document to specify the resulting implementation 
behavior in this case -- [RFC3720] already defines the error handling 
for this error scenario. 


10.3.12. Logout Command 
The Send Control Operational Primitive with the following input 


qualifier is used for requesting the transmission of a Logout Command 
PDU. 
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a) BHS of the Logout Command PDU as defined in [RFC3720]. 
10.3.13. Logout Response 
The Send Control Operational Primitive with the following input 
qualifier is used for requesting the transmission of a Logout 
Response PDU. 
a) BHS of the Logout Response PDU as defined in [RFC3720]. 
10.3.14. SNACK Request 
The Send Control Operational Primitive with the following input 
qualifier is used for requesting the transmission of a SNACK Request 
PDU. 
a) BHS of the SNACK Request PDU as defined in [RFC3720]. 


10.3.15. Reject 


The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a Reject PDU. 


a) BHS of the Reject PDU as defined in [RFC3720]. 
b) DataDescriptorReject: that defines the iSCSI Reject buffer. 
10.3.16.  NOP-Out 


The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a NOP-Out PDU. 


a) BHS of the NOP-Out PDU as defined in [RFC3720]. 
b) DataDescriptorNOPOut: that defines the iSCSI Ping data buffer. 
10.3.17. NOP-In 


The Send Control Operational Primitive with the following input 
qualifiers is used for requesting the transmission of a NOP-In PDU. 


a) BHS of the NOP-In PDU as defined in [RFC3720]. 


b) DataDescriptorNOPIn: that defines the iSCSI Return Ping data 
buffer. 
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4. Interactions for Receiving an iSCSI PDU 


The only PDUs that are received by an iSCSI layer operating on a 
Datamover layer are the iSCSI control-type PDUs. The Datamover layer 
delivers the iSCSI control-type PDUs as they arrive, qualifying each 
with the Connection Handle (see Section 5.3) that identifies the 
iSCSI connection for which the PDU is meant. The subsequent 
processing of the iSCSI control-type PDUs proceeds as defined in 
[RFC3720]. 


4.1. General Control-Type PDU Notification 


This sub-section describes the general mechanics applicable to 
several control-type PDUs. The following sub-sections note 
additional considerations for control-type PDUs that are not covered 
in this sub-section. 


The Control Notify Operational Primitive is used to notify the iSCSI 
layer of the arrival of the following iSCSI control-type PDUs: SCSI 
Command, SCSI Response, Task Management Function Request, Task 
Management Function Response, Asynchronous Message, Text Request, 
Text Response, Logout Command, Logout Response, SNACK, Reject, NOP- 
Out, NOP-In. 


4.2. SCSI Data Transfer PDUs 
4.2.1. SCSI Data-Out 


The Control Notify Operational Primitive is used to notify the iSCSI 
layer of the arrival of a SCSI Data-Out PDU carrying the non- 
immediate unsolicited data. Note however that the solicited SCSI 
Data-Out arriving on the target does not cause a notification to the 
iSCSI layer using the Control Notify Primitive because the solicited 
SCSI Data-Out was not sent by the initiator iSCSI layer as control- 
type PDUs. 


4.2.2. SCSI Data-In 


The Datamover layer does not notify the iSCSI layer of the arrival of 
the SCSI Data-in at the initiator, because SCSI Data-in is an iSCSI 
data-type PDU (see section 5.1). The iSCSI layer at the initiator 
however may infer the arrival of the SCSI Data-In when it receives a 
subsequent notification of the SCSI Response PDU via a Control Notify 
invocation. 


While this document does not contemplate the possibility of a Data-In 
PDU being received at the initiator iSCSI layer, specific Datamover 
protocols may define how to deal with an unexpected inbound SCSI 
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Data-In PDU that may result in the initiator iSCSI layer receiving 
the Data-In PDU. This document leaves the details of handling this 
error scenario to the specific Datamover protocols, so each may 
define the appropriate error handling specific to the Datamover 
environment. 


4.2.3. Ready To Transfer (R2T) 


Because an R2T PDU is an iSCSI data-type PDU (see Section 5.1) that 
is not delivered as-is to the initiator iSCSI layer, the Datamover 
layer does not notify the iSCSI layer of the arrival of an R2T PDU. 
When an iSCSI node sends an R2T PDU to its local Datamover layer, the 
local and remote Datamover layers transparently bring about the data 
transfer requested by the R2T PDU. 


While this document does not contemplate the possibility of an R2T 
PDU being received at the initiator iSCSI layer, specific Datamover 
protocols may define how to deal with an unexpected inbound R2T PDU 
that may result in the initiator iSCSI layer receiving the R2T PDU. 
This document leaves the details of handling this error scenario to 
the specific Datamover protocols, so each may define the appropriate 
error handling specific to the Datamover environment. 


4.3. Login Request 


The Control Notify Operational Primitive is used for notifying the 
target iSCSI layer of the arrival of a Login Request PDU. Note that 
Specific Datamover protocols may choose to disallow the standard DA 
Primitives from being used for the iSCSI Login Phase. When used in 
conjunction with such Datamover protocols, the arrival of a Login 
Request necessitating the Control Notify Operational Primitive 
invocation is clearly an error scenario, as the Login Request PDU is 
arriving in the iSCSI Full Feature Phase. It is outside the scope of 
this document to specify the resulting implementation behavior in 
this case -- [RFC3720] already defines the error handling in this 
error scenario. 


4.4. Login Response 


The Control Notify Operational Primitive is used to notify the 
initiator iSCSI layer of the arrival of a Login Response PDU. Note 
that specific Datamover protocols may choose to disallow the standard 
DA Primitives from being used for the iSCSI Login Phase. When used 
in conjunction with such Datamover protocols, the arrival of a Login 
Response necessitating the Control Notify Operational Primitive 
invocation is clearly an error scenario, as the Login Response PDU is 
arriving in the iSCSI Full Feature Phase. It is outside the scope of 
this document to specify the resulting implementation behavior in 
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this case -- [RFC3720] already defines the error handling in this 
error scenario. 


11. Security Considerations 
11.1. Architectural Considerations 


DA enables compliant iSCSI implementations to realize a control and 
data separation in the way they interact with their Datamover 
protocols. Note however that this separation does not imply a 
separation in transport mediums between control traffic and data 
traffic -- the basic iSCSI architecture with respect to tasks and PDU 
relationships to tasks remains unchanged.  [RFC3720] defines several 
MUST requirements on ordering relationships across control and data 
for a given task besides a mandatory deterministic task allegiance 
model -- DA does not change this basic architecture (DA has a 
normative reference to [RFC3720]) for allow any additional 
flexibility in compliance in this area. To summarize, sending bulk 
data transfers (prompted by Put Data and Get Data Primitive 
invocations) on a different transport medium would be as ill-advised 
as sending just the Data-Out/Data-In PDUs on a different TCP 
connection in RFC 3720-based iSCSI implementations. Consequently, 
all the iSCSI-related security text in [RFC3723] is directly 
applicable to a DA-enabled iSCSI implementation. 


Another area with security implications is the Datamover connection 
resource management model, which DA defines -- particularly the 
Allocate Connection Resources Primitive. An inadvertent realization 
of this model could leave an iSCSI implementation exposed to denial- 
of-service attacks. As Figures 2 and 3 in Section 13.2 illustrate, 
the most effective countermeasure to this potential attack consists 
of performing the Datamover resource allocation when the iSCSI layer 
is sufficiently far along in the iSCSI Login Phase that it is 
reasonably certain that the peer side is not an attacker. In 
particular, if the Login Phase includes a SecurityNegotiation stage, 
an iSCSI end node MUST defer the Datamover connection resource 
allocation (i.e., invoking the Allocate Connection Resources 
Primitive) to the LoginOperationalNegotiation stage [RFC3720] so that 
the resource allocation happens post-authentication. This 
considerably minimizes the potential for a denial-of service attack. 


11.2. Wire Protocol Considerations 


In view of the fact that the DA architecture itself does not define 
any new wire protocol or propose modifications to the existing 
protocols, there are no additional wire protocol security 
considerations in employing DA itself. However, a DA-compliant iSCSI 
implementation MUST comply with all the iSCSI-related requirements 
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Stipulated in [RFC3723] and [RFC3720]. Note further that in 
realizing DA, each Datamover protocol must define and elaborate as 
appropriate on any additional security considerations resulting from 
the use of that Datamover protocol. 


All Datamover protocol designers are strongly recommended to refer to 
[RDDPSEC] for the types of security issues to consider. While 
[RDDPSEC] elaborates on the security considerations applicable to an 
RDDP-based Datamover [iSER], the document is representative of the 
type of analysis of resource exhaustion and the application of 
countermeasures that need to be done for any Datamover protocol. 
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Appendix A. Design Considerations and Examples 
A.1. Design Considerations for a Datamover Protocol 


This section discusses the specific considerations for RDMA-based and 
RDDP-based Datamover protocols. 


a) Note that the modeling of interactions for SCSI Data-Out 
(Section 10.3.5.1) is only used for unsolicited data transfer. 


b) The modeling of interactions for SNACK (Sections 10.3.14 and 
10.4.1) is not expected to be used given that one of the design 
requirements on the Datamover is that it "guarantees an error- 
free, reliable, in-order transport mechanism" (Section 6). The 
interactions for sending and receiving a SNACK are nevertheless 
modeled in this document because the receiving iSCSI layer can 
deterministically deal with an inadvertent SNACK. This also 
shows the DA designers' intent that DI is not meant to filter 
certain types of PDUs. 


c) The onus is on a reliable Datamover (per requirements stated in 
Section 6) to realize end-to-end data acknowledgements via 
Datamover-specific means. In view of this, even use of data- 
ACK-type SNACKs are unnecessary. Consequently, an initiator 
may never request sending a SNACK Request in this model 
assuming that the proactive (timeout-driven) SNACK 
functionality is turned off in the legacy iSCSI code. 


d) Note that the current DA model for bootstrapping a 
Connection Handle into service -- i.e., associating a new iSCSI 
connection with a Connection Handle -- clearly implies that the 
iSCSI connection must already be in Full Feature Phase when the 
Datamover layer comes into the stack. This further implies 
that the iSCSI Login Phase must be carried out in the 
traditional "Byte streaming mode" with no assistance or 
involvement from the Datamover layer. 


A.2. Examples of Datamover Interactions 
The figures described in this section provide some examples of the 
usage of Operational Primitives in interactions between the iSCSI 
layer and the Datamover layer. The following abbreviations are used 
in this section. 


Avail - Available 


Abted - Aborted 
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Buf - I/O Buffer 

Cmd - Command 

Compl - Complete 

Conn - Connection 

Ctrl Ntfy - Control Notify 

Dal Tk Res - Deallocate Task Resources 
Data Cmp Nfy - Data Completion Notify 
Data ACK Nfy - Data ACK Notify 

DM - Datamover 

Imm - Immediate 

Snd Ctrl - Send Control 

Msg - Message 

Resp - Response 

Sol - Solicited 

TMF Req - Task Management Function Request 
TMF Res - Task Management Function Response 
Trans - Transfer 


Unsol - Unsolicited 
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Figure 3. A Successful iSCSI Login on 
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Figure 6.  iSCSI Does Not Enable the Datamover 
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Figure 8. An Abnormal iSCSI Connection Termination 
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Figure 11. A SCSI Read Data Acknowledgement 
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